This PR adds a new feature: The query latency details will be tracked and reported; the index size will be also reported.

Changes:

• modified: examples/inference/distilsplade_max/beir_scifact/all_in_one.sh: Check out this example for the final demo report!
• modified: sparse_retrieval/inference/aio.py: Add new arguments to search.run and evaluate.run;
• modified: sparse_retrieval/inference/search.py:
  • Add new argument output_latency to specify the output latency file.
  • Add a LatencyReporter;
  • Use the LatencyReporter to record each searcher.search/batch_search and report the latency details into a file called latency.tsv (under the same path of run.tsv);
    • Each line of latency.tsv is f"{qid}\t{word_length}\t{latency}\t{batch_size}\n"
• modified: sparse_retrieval/inference/evaluate.py:
  • Add new argument latency_path and index_path as the paths to the stats source;
  • Add new argument bins to specify the binning query latencies wrt. how many word-length bins.
  • Summarize and report latency details:

      latency_info = {
        "latency": {
            "latency_avg": np.mean(latencies),
            "query_word_length_avg": np.mean(word_lengths),
            "binned": {
                "word_length_bins": word_length_bins.tolist(),
                "freqs": freqs.tolist(),
                "latencies_avg": binned_latencies_avg,
                "latencies_std": binned_latencies_std
            },
            "batch_size": np.mean(batch_sizes),
            "processor": get_processor_name()
        }
    }

  • Report index size in MB.
• modified: sparse_retrieval/inference/utils.py: Some new util functions to support the new features.